Holographic display apparatus including freeform curved surface and operating method thereof

ABSTRACT

A holographic display apparatus including a freeform curved surface and an operating method of the holographic display apparatus are provided. The holographic display apparatus includes: an image generator configured to generate a hologram image by modulating light; an optical system including a freeform curved surface for forming the hologram image generated by the image generator in a predetermined depth; and a processor configured to generate a computer-generated hologram (CGH) based on three-dimensional image information by using a phase map including information about an optical aberration with respect to the freeform curved surface and to control the image generator to modulate the light based on the CGH.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Applications No. 10-2021-0053105, filed on Apr. 23,2021, in the Korean Intellectual Property Office, the disclosure ofwhich is incorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to a holographic display apparatus including afreeform curved surface and an operating method of the holographicdisplay apparatus.

2. Description of Related Art

A head-mounted display (HMD) providing virtual reality (VR) has beenrecently commercialized and widely used in the entertainment industry.Also, the HMD has been developed into a form to be applied to medical,educational, and industrial fields.

An augmented reality (AR) display, which is an advanced form of a VRdisplay, is an image apparatus that combines a real world with VR toderive an interaction between the real world and the VR. The interactionbetween the real world and the VR is based on a function of providinginformation about a real situation in real time, and by overlapping avirtual object or information on a real world environment, an effect ofthe real world may further be increased.

SUMMARY

Provided are a holographic display apparatus having a freeform curvedsurface and an operating method of the holographic display apparatus.

Provided are a holographic display apparatus capable of compensating fordistortion due to a freeform curved surface and an operating method ofthe holographic display apparatus.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of embodiments of the disclosure.

In accordance with an aspect of the disclosure, a holographic displayapparatus includes an image generator configured to generate a hologramimage by modulating light; an optical system including a freeform curvedsurface configured to form the hologram image generated by the imagegenerator at a predetermined depth; and a processor configured to:generate a computer-generated hologram (CGH) based on three-dimensionalimage information using a phase map, the phase map including informationabout an optical aberration due to the freeform curved surface; andcontrol the image generator to modulate the light based on the CGH.

The phase map may include phase information of the light based on alight path on which the light modulated by the image generator istransmitted to a viewer by the freeform curved surface.

The phase information of the light may include a difference between aphase of the light at a time when the light is modulated by the imagegenerator and a phase of the light at a time when the light reaches theviewer.

The phase information may be inversely proportional to a wavelength ofthe light modulated by the image generator.

The phase information may be proportional to a refractive index of amedium on the light path on which the light modulated by the imagegenerator is transmitted to the viewer through the freeform curvedsurface.

The phase information may be proportional to a length of the light pathon which the light modulated by the image generator is transmitted tothe viewer through the freeform curved surface.

The processor may be further configured to perform phase modulation onthe CGH using the phase map.

The processor may be further configured to modulate the CGH using aconjugate value of the phase map.

The processor may be further configured to apply, to thethree-dimensional image information, a distortion compensation algorithmconfigured to compensate for image shape distortion due to the freeformcurved surface.

The distortion compensation algorithm may include an inverselytransformed algorithm of a mapping algorithm, the mapping algorithmbeing configured to map location information of pixels included in thethree-dimensional image information with location information of pixelsof a virtual image corresponding to the three-dimensional imageinformation.

The processor may be further configured to apply the distortioncompensation algorithm to an entirety of the location information of thepixels included in the three-dimensional image information.

The processor may be further configured to apply the distortioncompensation algorithm to the location information of one or more pixelsfrom among the pixels included in the three-dimensional imageinformation and linearly interpolate the location information of theother pixels from among the pixels included in the three-dimensionalimage information.

The optical system may include a combiner configured to converge thehologram image and external light to one point, the external lightcorresponding to an external environment, and the freeform curvedsurface may be integral with the combiner.

The combiner may include a waveguide configured to transmit the hologramimage, wherein the freeform curved surface is arranged on a surface ofthe waveguide.

The combiner may further include a transflective layer arranged on thefreeform curved surface.

The holographic display apparatus may include a virtual realityapparatus or an augmented reality apparatus.

In accordance with an aspect of the disclosure, an operating method of aholographic display apparatus including a freeform curved surfaceincludes generating a computer-generated hologram (CGH) based onthree-dimensional image information using a phase map, the phase mapincluding information about an optical aberration due to the freeformcurved surface; generating a hologram image by modulating light based onthe CGH; and forming the hologram image at a predetermined depth by thefreeform curved surface.

The phase map may include phase information of the light based on alight path on which the light modulated as the hologram image istransmitted to a viewer by the freeform curved surface.

The phase information of the light may include a difference between aphase of the light at a time when the light is modulated as the hologramimage and a phase of the light at a time when the light reaches theviewer.

The generating of the CGH may include modulating the CGH using aconjugate value of the phase map.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a schematic diagram of components of a holographic displayapparatus applied to virtual reality (VR), according to an embodiment;

FIG. 2 is a schematic diagram of components of an image generator ofFIG. 1;

FIG. 3 is a schematic diagram of components of a holographic displayapparatus applied to augmented reality (AR), according to an embodiment;

FIG. 4 is a flowchart of a method of calculating distortion of a virtualimage according to an optical aberration of a freeform curved surface,according to an embodiment;

FIG. 5 is a reference diagram for describing a portion of a phase mapusing a ray, according to an embodiment;

FIG. 6 is a flowchart for describing a method of generating a hologramimage by using a phase map including information about an opticalaberration of a freeform curved surface, according to an embodiment;

FIG. 7 is a reference diagram for describing, based on a wavefront,compensation of an optical aberration due to a freeform curved surface,according to an embodiment;

FIGS. 8A through 8C are reference diagrams for describing an algorithmfor compensating for shape distortion of an image, according to anembodiment;

FIG. 9A illustrates a result of simulating a virtual image by using afreeform curved surface;

FIG. 9B illustrates a result of simulating a virtual image by using aphase map and a distortion compensation algorithm, according to anembodiment;

FIG. 10 illustrates a display apparatus configured to provide an imageto each of both eyes, according to an embodiment;

FIG. 11 illustrates an example in which a display apparatus according toan embodiment is applied to a vehicle; and

FIG. 12 illustrates an example in which a display apparatus according toan embodiment is applied to AR glasses or VR glasses.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings, wherein like referencenumerals refer to like elements throughout. In this regard, embodimentsmay have different forms and should not be construed as being limited tothe descriptions set forth herein. Accordingly, embodiments are merelydescribed below, by referring to the figures, to explain aspects. Asused herein, the term “and/or” includes any and all combinations of oneor more of the associated listed items. Expressions such as “at leastone of,” when preceding a list of elements, modify the entire list ofelements and do not modify the individual elements of the list.

Hereinafter, a display apparatus including a freeform curved surfacewill be described in detail by referring to the accompanying drawings.In the drawings, the same reference numerals denote the same elementsand sizes of elements may be exaggerated for clarity and convenience ofexplanation. Also, embodiments described hereinafter are only examples,and various modifications may be made based on the embodiments.

Hereinafter, it will be understood that when an element is referred toas being “on” or “above” another element, the element can be directlyover or under the other element and directly on the left or on the rightof the other element, or intervening elements may also be presenttherebetween. As used herein, the singular terms “a” and “an” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that when a part“includes” or “comprises” an element, unless otherwise defined, the partmay further include other elements, not excluding the other elements.

The term “the” and other equivalent determiners may correspond to asingular referent or a plural referent. Operations included in a methodmay be performed in an appropriate order, unless the operations includedin the method are described to be performed in an apparent order, orunless the operations included in the method are described to beperformed otherwise.

Also, the terms such as “ . . . unit,” “module,” or the like used in thespecification indicate a unit, which processes at least one function ormotion, and the unit may be implemented by hardware or software, or by acombination of hardware and software.

The connecting lines, or connectors shown in the various figurespresented are intended to represent example functional relationshipsand/or physical or logical couplings between the various elements. Itshould be noted that many alternative or additional functionalrelationships, physical connections or logical connections may bepresent in a practical device.

Operations included in a method may be performed in an appropriateorder, unless the operations included in the method are described to beperformed in an apparent order, or unless the operations included in themethod are described to be performed otherwise. The operations are notnecessarily limited to the described order.

The use of all examples and example terms are merely for describing thedisclosure in detail and the disclosure is not limited to the examplesand the example terms.

FIG. 1 is a schematic diagram of components of a holographic displayapparatus 10 applied to virtual reality (VR), according to anembodiment, and FIG. 2 is a schematic diagram of components of an imagegenerator 110 of FIG. 1. Referring to FIG. 1, the holographic displayapparatus 10 according to an embodiment may include the image generator110 configured to generate a hologram image, an optical system 120including a freeform curved surface 121 for forming the hologram imageat a predetermined depth D, and a processor 130 configured to generate acomputer-generated hologram (CGH) based on three-dimensional imageinformation and provide the CGH to the image generator 110.

The image generator 110 may generate the hologram image by modulatinglight. As illustrated in FIG. 2, the image generator 110 may include alight source 210 configured to provide coherent light, a spatial lightmodulator 220 configured to generate the hologram image by diffractingthe light from the light source 210, and a focal optical system 230configured to form the hologram image on a predetermined space.

The light source 210 may provide coherent light. The light source 210may include a laser diode. However, when light has a predetermineddegree of spatial coherence, the light may become coherent by beingdiffracted and modulated by the spatial light modulator 220, and thus,other light sources emitting light having a predetermined degree ofspatial coherence may also be used.

Other elements may be further arranged, including for example, awaveguide configured to transmit light incident from the light source210 and to output light of a different dimension, and a beam expanderlocated between the light source 210 and the waveguide and configured toexpand a dimension of light.

The spatial light modulator 220 may generate the hologram image bydiffracting incident light. A hologram method uses a principle that whena reference wave is irradiated to a hologram recording an interferencefringe between an object wave and the reference wave, the object wave isreproduced. Recently, to form this interference fringe, a CGH has beenused.

The spatial light modulator 220 may include, for example, a liquidcrystal-on-silicon (LCoS) device, a liquid crystal display (LCD), anorganic light-emitting diode (OLED), a digital micro-mirror device(DMD), and next-generation display devices, such as amicro-light-emitting diode (micro-LED), a quantum dot (QD) LED, etc.

The focal optical system 230 may be configured to display the hologramimage according to depth information included in the hologram image.That is, the focal optical system 230 may be configured to representmultiple depths, and through the multiple depth representation, visualfatigue may be reduced. The focal optical system 230 may be configuredto display the hologram image generated by the spatial light modulator220 on a hologram plane HP and may vary a location of the hologram planeHP.

The focal optical system 230 may include one or more lenses. The one ormore lenses may be configured to have varied curvatures or may beconfigured to move in an optical axis direction, and thus, the locationof the hologram plane HP, on which the hologram image is displayed, maybe varied.

The processor 130 may generate the CGH based on the three-dimensionalimage information. Also, the processor 130 may compensate for distortionof the hologram image by modulating a phase of the CGH.

According to a method of generating the CGH, the CGH may be divided intoa point cloud type, a polygonal type, a depth map (or a layer-based)type, etc. However, the method of generating the CGH is well known, andthus, its detailed description is omitted.

The processor 130 may correspond to processors included in various typesof computing devices, such as a personal computer (PC), a server device,a television (TV), a mobile device (a smart phone, a tablet device,etc.), an embedded device, an autonomous vehicle, a wearable device, anaugmented reality (AR) device, an Internet of Things (IoT) device, etc.For example, the processor 130 may correspond to processors, such as acentral processing unit (CPU), a graphics processing unit (GPU), anapplication processor (AP), a neural processing unit (NPU), etc., but isnot limited thereto.

Generally, when the CGH is input to the spatial light modulator 220, thespatial light modulator 220 may modulate the incident light according tothe CGH and generate the hologram image. To generate the CGH, a hologramvalue with respect to each location of the hologram plane HP may begenerated. To this end, a fast Fourier transform (FFT) may be generatedfor each layer to generate an appropriate focal point for all depthplanes on a space. Because the CGH has to be generated with respect toall layers, the amount of generation is large.

Thus, to simplify processing of the three-dimensional image information,the processor 130 may determine a representative depth d from thethree-dimensional image information. The processor 130 may determine therepresentative depth d by analyzing color information and depthinformation included in the three-dimensional image information. Therepresentative depth d may be connected with a depth at which thefreeform curved surface to be described below forms a virtual image.

The image information may include the color information and the depthinformation for each of images in a frame unit. In other words, theimage information may include a single depth information for each frame.When the depth information included in the image information is depthinformation in the frame unit, the processor 130 may generate the CGHbased on the depth information.

When the depth information is not the depth information in the frameunit, for example, when the depth information is provided in units of asub-image or a pixel, the processor 130 may determine a representativedepth d of the frame unit and may generate the CGH based on therepresentative depth d.

The processor 130 may determine the representative depth d with respectto each image in the frame unit by using the color information and/orthe depth information included in the image information.

For example, the processor 130 may extract a color map from the imageinformation and may perform content analysis and/or saliency informationanalysis on the color map to determine the representative depth d. Thesaliency information analysis may be performed to determine an areahaving a high probability of being viewed by a viewer, that is, an areahaving a high visual concentration degree. In order to determine thearea having the high visual concentration degree, brightness, a color,an outline, an object size, etc. may be taken into account. For example,an area having a large difference in brightness or color from an ambientenvironment, having a distinct outline, or having a large-sized objectmay be the area having the high visual concentration degree. A depthvalue corresponding to this area may be determined as the representativedepth d. Alternatively, according to content of an image, a locationhaving a high visual concentration degree may be determined.

In addition, the processor 130 may determine the representative depth dby taking into account a zone of comfort from the depth map and mayquantize depth information included in the depth map to determine therepresentative depth d based on the quantized depth information.

The processor 130 may generate the CGH according to the determinedrepresentative depth d.

That is, the processor 130 may obtain one CGH image layer. The processor130 may generate the CGH having the representative depth d for eachframe of the three-dimensional image information. Thus, the spatiallight modulator 220 may generate the hologram image according to theCGH, and the focal optical system 230 may display the generated hologramimage on the hologram plane HP located at the representative depth d. Itis described that the focal optical system 230 may adjust the focalpoint to form the hologram plane HP at the representative depth d.However, the disclosure is not limited thereto. By changing a locationof the spatial light modulator 220, the hologram plane HP correspondingto the representative depth d may be adjusted.

The freeform curved surface 121 may form a virtual image correspondingto the hologram image at a predetermined depth D corresponding to therepresentative depth d, that is, the predetermined depth D from an eyeof a user. In detail, the hologram image generated by the imagegenerator 110 may be reflected from the freeform curved surface 121 andtransmitted to the eye of the user. The viewer may recognize that thehologram image is formed at the predetermined depth D based on a lengthof a light path between the hologram image generated by the imagegenerator 110 and the freeform curved surface 121. Since the viewer mayrecognize a hologram image located in a different location from theimage generated by the image generator 110, the image recognized by theviewer may be referred to as a virtual image VI.

The freeform curved surface 121 refers to a curved surface that isoptimally designed to focus off-axially incident light to one focalpoint or to wholly form an obliquely incident image with respect to anoptical axis. In general, a mirror surface having a curvature maysoundly operate with respect to light that is incident based on anoptical axis, but may have a significant optical aberration with respectto obliquely incident light.

When a display apparatus applied to VR or AR is developed, in order torelatively more freely arrange a location of the image generator 110while reducing a size of the overall system, an optical system may haveto be able to form an obliquely incident image as a sound virtual image.In this case, instead of a general mirror surface, the freeform curvedsurface 121, which is optimally designed with respect to the imagegenerator 110 in a fixed location, may be used as illustrated in FIG. 1to form a high quality virtual image.

FIG. 3 is a schematic diagram of components of a holographic displayapparatus 20 applied to AR, according to an embodiment. To compare FIGS.1 and 3, the holographic display apparatus 20 of FIG. 3 may include theimage generator 110 configured to generate a hologram image, a combiner120 a that is an optical system configured to mix a virtual image andlight containing an external scenery and provide the mixed image to aviewer, and a processor 130 configured to generate a CGH based onthree-dimensional image information and provide the CGH to the imagegenerator 110.

The combiner 120 a may transmit not only light L₁ containing an imagegenerated by the image generator 110, but also light L₂ containing anexternal scenery of a front side of a viewer, to an eye of the viewer.For example, the combiner 120 a may reflect the light L₁ containing theimage toward the eye of the viewer and may transmit the L₂ containingthe external scenery toward the eye of the viewer.

The external light L₂ may contain a real scenery of the front side ofthe viewer, rather than an image generated by the image generator 110.Thus, the viewer may simultaneously recognize the image artificiallygenerated by the image generator 110 and the real front view. Thus, theholographic display apparatus 20 may function as a see-through typedisplay.

The combiner 120 a may include a waveguide 122 configured to transmitthe image generated by the image generator 110. The waveguide 122 mayinclude a plurality of surfaces, and at least one of the plurality ofsurfaces may include the freeform curved surface 121.

As illustrated in FIG. 3, the waveguide 122 may include a first surfaceS1 and a second surface S2 that is the freeform curved surface 121,which are arranged to face each other, and a third surface S3 and afourth surface S4 arranged to face each other between the first andsecond surfaces S1 and S2. It is illustrated that the second surface S2is the freeform curved surface 121. However, the disclosure is notlimited thereto. The first surface S1 may also be the freeform curvedsurface. The third surface S3 and the fourth surface S4 may be arrangedto be parallel to each other so as not to have refractive power.

The combiner 120 a may further include a light transmissive plate 123contacting the waveguide 122. The light transmissive plate 123 mayinclude a curved surface having a complementary shape with respect tothat of the freeform curved surface 121, and may share the third surfaceS3 and the fourth surface S4 of the waveguide 122. The light L₂containing the external scenery may be incident into the fourth surfaceS4, and then, may sequentially pass through the freeform curved surface121 and the third surface S3 so as to be incident to the eye of theviewer.

A transflective layer may be arranged on the freeform curved surface 121to reflect the light L₁ containing the image and transmit the light L₂containing the external scenery. The transflective layer may simplyreflect a portion of incident light and transmit a remaining portion ofthe incident light. In this case, a portion of the light L₁ containingthe image may be reflected by the transflective layer of the freeformcurved surface 121 to proceed toward the eye of the viewer, and aportion of the light L₂ containing the external scenery may betransmitted by the transflective layer of the freeform curved surface121 to proceed toward the eye of the viewer.

When the light L₁ containing the image generated by the image generator110 has polarization characteristics, the transflective layer may beconfigured to reflect light having predetermined polarization propertiesand transmit light having other polarization properties. For example,when the light L₁ containing the virtual image has a first polarizationproperty, the transflective layer may reflect light having the firstpolarization property and may transmit light having a secondpolarization property perpendicular to the first polarization property.

In FIG. 3, it is described that the transflective layer is arranged onthe freeform curved surface 121. However, the disclosure is not limitedthereto. The transflective layer and the freeform curved surface 121 mayalso be separately arranged on the waveguide 122.

The freeform curved surface 121 described above may be optimallydesigned with respect to the location of the image generator 110. Forexample, a profile of the freeform curved surface 121 may be designedthrough an optimization process for satisfying various conditions, suchas a viewing angle of a virtual image recognized by a viewer, athickness of the optical system 120 including the freeform curvedsurface 121, a size of an eye box, a location of the image generator110, etc.

Just when an optical condition applied to design the freeform curvedsurface 121 and an optical condition applied to use the freeform curvedsurface 121 are slightly different from each other, the viewer mayrecognize a virtual image having a deteriorated quality, that is, avirtual image having a large optical aberration or distortion.Alternatively, when the freeform curved surface 121 is not optimallydesigned, it may be difficult to realize a high quality virtual image.The freeform curved surface 121 is designed with respect to the opticalsystem 120 that is off-axial, and thus, distortion occurring due to thefreeform curved surface 121 may become much more irregular andsignificant than pincushion distortion occurring when a general lens isused to view a virtual image. Thus, there is a limitation to theapplication of an algorithm for compensating for distortion of an image,which is generally used in a previous display apparatus.

FIG. 4 is a flowchart of a method of calculating distortion of a virtualimage according to an optical aberration of the freeform curved surface121, according to an embodiment, and FIG. 5 is a reference diagram fordescribing a method of obtaining a phase map by using a ray, accordingto an embodiment. As illustrated in FIG. 4, the processor 130 may samplea virtual image delivered to a viewer into a plurality of rays for eachfield (S310). An image sensor, for example, a charge-coupled device(CCD) sensor, may be arranged on a space on which an eye of the vieweris located, and the processor 130 may sample the virtual image into theplurality of rays for each field by using a result received from thesensor.

The processor 130 may calculate a light path of each of the plurality ofrays backwardly propagated to the image generator 110 through thefreeform curved surface 121 (S320). The light path may be calculated byusing various optical software, such as Zemax, CodeV, Light tools, etc.

The processor 130 may calculate a phase difference of each ray on thelight path as shown in the following Equation 1 (S330).

$\begin{matrix}{{\varnothing_{k} = {\frac{2\pi}{\lambda} \cdot n \cdot d_{k}}},} & \left\lbrack {{Equation}1} \right\rbrack\end{matrix}$

Here, ø_(k) indicates a phase difference of a k^(th) ray of a field of apredetermined direction, A indicates a wavelength of the ray, nindicates a refractive index, and d_(k) indicates a length of the lightpath of the k^(th) ray. In detail, a phase difference of a ray denotes adifference between a phase of light at a time when the light ismodulated by the image generator 110 and a phase of light at a time whenthe light is transmitted to the viewer (e.g., at a time when the lightreaches the viewer), the refractive index denotes valid refractiveindices of all mediums arranged on the light path where the raysmodulated by the image generator 110 are transmitted to the viewerthrough the freeform curved surface 121, and the length of the lightpath denotes lengths of all light paths, on which the rays modulated bythe image generator 110, are transmitted to the viewer through thefreeform curved surface 121.

As illustrated in Equation 1, the phase difference of each ray may beinversely proportional to the wavelength of light, may be proportionalto the refractive index of the medium on the light path, and may beproportional to the length of the light path.

The processor 130 may obtain a phase map based on the phase difference(S340). In detail, the processor 130 may obtain phase information (thatis, wavefront information) of a corresponding field by sequentiallyperforming interpolation by applying a point spread function to thephase differences of the rays included in each field. Also, theprocessor 130 may obtain the phase map by summing wavefront informationof the fields with respect to the entire virtual image.

FIG. 5 is a diagram of a portion of a phase map obtained from a ray of apredetermined field, according to an embodiment. When the freeformcurved surface 121 is ideally designed, a field in a predetermineddirection may be converged to a phase value of a predetermined pixel.However, when the freeform curved surface 121 is not optimally designed,or due to an error in the arrangement between the optical system 120 andthe image generator 110 of the holographic display apparatus 10, theremay be phase information 400 as illustrated in FIG. 5. The phaseinformation 400 described above may be a result of applying the pointspread function to the phase differences of the rays included in apredetermined field.

The phase map may be obtained when the holographic display apparatus 10is manufactured and may be pre-stored in a memory of the holographicdisplay apparatus 10. However, the phase map is not limited thereto.After arranging a sensor (for example, camera) corresponding to an eyeof the viewer at a location at which the eye of the viewer is to bepositioned, in the holographic display apparatus 10 which ismanufactured, the processor 130 of the holographic display apparatus 10may obtain the phase map. Alternatively, the processor 130 of theholographic display apparatus 10 may update a pre-stored phase map.

The image generator 110 may generate the hologram image by adjusting anamplitude and a phase of light, and thus, the holographic displayapparatus 10 according to an embodiment may provide a virtual image inwhich an optical aberration generated due to the freeform curved surface121 is compensated for by using the phase map.

FIG. 6 is a flowchart for describing a method of generating a hologramimage by using a phase map including information about an opticalaberration of a freeform curved surface, according to an embodiment.

The processor 130 may generate a CGH based on three-dimensional imageinformation (S510). According to a method of generating the CGH, the CGHmay be divided into a point cloud type, a polygonal type, a depth map(or a layer-based) type, etc. However, the method of generating the CGHis well known, and thus, its detailed description is omitted.

The processor 130 may modulate the CGH by using the phase map includingthe information about the optical aberration of the freeform curvedsurface 121 and may apply the modulated CGH to the image generator 110(S520). The phase map may include the information about the opticalaberration of the freeform curved surface, and thus, the processor 130may generate the modulated CGH by applying a conjugate value of thephase map to the CGH.

The image generator 110 may generate the hologram image by modulatingthe light using the modulated CGH (S530). The hologram image may bereflected from the freeform curved surface 121 and may be incident to aneye of a viewer. The hologram image may be generated based on the CGHmodulated using the conjugate value of the phase map, and thus, whenlight corresponding to the hologram image is reflected by the freeformcurved surface 121, the light may be converted to light in which theoptical aberration due to the freeform curved surface 121 is compensatedfor and may be incident to the eye of the viewer.

FIG. 7 is a reference diagram for describing, based on a wavefront,compensation of an optical aberration due to the freeform curved surface121, according to an embodiment. As illustrated in FIG. 7, the imagegenerator 110 may generate the hologram image by using the CGH modulatedusing the phase map including the information about the opticalaberration of the freeform curved surface 121, and thus, a hologramimage HI′ generated by the image generator 110 may include not onlycolor information and depth information, but also optical aberrationinformation. The hologram image HI′ may be reflected by the freeformcurved surface 121, and thus, the optical aberration information may beoffset, and a hologram image HI including color information and depthinformation may be incident to the eye of the viewer.

The freeform curved surface 121 may be designed for an optical system120 that is off-axial, and thus, shape distortion of an image occurringdue to the freeform curved surface 121 may become much more irregularand significant than pincushion distortion occurring when a generalconvex lens is used to view a virtual image. For example, even when onepixel is generated by compensating for a phase by backwardly propagatinga central field of the virtual image, a location to which correspondingrays are converged may not be the center of the spatial light modulator220. This error may be increased toward an outer portion of an image,that is, this error may be increased as an incident angle of a field isincreased.

FIGS. 8A through 8C are reference diagrams for describing an algorithmfor compensating for shape distortion of an image, according to anembodiment. As illustrated in FIG. 8A, the processor 130 may obtain amapping algorithm configured to map pixel location information of anoriginal image (or may be referred to as “image information) and pixellocation information of a virtual image in a one-on-one correspondencemanner. To obtain the mapping algorithm, an image sensor may be arrangedat a location of a display apparatus, at which an eye of a viewer is tobe positioned, to obtain the virtual image. Also, the processor 130 mayobtain a distortion compensation algorithm by inversely transforming themapping algorithm.

Also, as illustrated in FIG. 8B, the processor 130 may apply thedistortion compensation algorithm to the original image based on thepixel information included in the image information and may generate aCGH based on the original image (e.g., the image information) to whichthe distortion compensation algorithm is applied, and the spatial lightmodulator 220 may generate the hologram image based on the CGH. Thehologram image may be an image based on the distortion compensationalgorithm.

As illustrated in FIG. 8C, in the hologram image, shape distortion of animage due to the reflection through the freeform curved surface 121 maybe compensated for, and a virtual image in which the shape distortion iscompensated for may be incident to the eye of the viewer.

The mapping algorithm may be obtained based on a relationship of pixelunits between the virtual image and the original image (e.g., the imageinformation). In other words, the mapping algorithm may be obtained tomap each pixel of the original image to a respective pixel of thevirtual image. However, the disclosure is not limited thereto. Themapping algorithm may be obtained based on a relationship between one ormore pixels of the virtual image and one or more corresponding pixels ofthe original image, and the distortion compensation algorithm may bebased on location information of the one or more pixels. Also, theprocessor 130 may obtain compensated image information by applying thedistortion compensation algorithm to one or more pixels of the originalimage (or the image information) and then applying linear interpolationto the rest of the pixels of the original image.

FIG. 9A illustrates a result of simulating a virtual image by using thefreeform curved surface 121, and FIG. 9B illustrates a result ofsimulating a virtual image by using a phase map and a distortioncompensation algorithm, according to an embodiment. As illustrated inFIG. 9A, it is identified that before compensation, an image isdistorted, and a virtual image that is not vivid due to an opticalaberration is provided. However, after applying the phase map and thedistortion compensation algorithm, it is identified as illustrated inFIG. 9B that not only the image is not distorted, but also the vividimage is provided.

FIG. 10 illustrates a display apparatus 60 configured to provide animage to each of both eyes, according to an embodiment. The imageprovided to each of the eyes may be the same or may be an image havingtime difference information.

The display apparatus 60 may include a first image generator 110Rconfigured to generate an image for a right eye, a first imageconvergence member 120R configured to converge the image for the righteye and an image of a real environment to one area, a second imagegenerator 110L configured to generate an image for a left eye, a secondimage convergence member 120L configured to converge the image for theleft eye and an image of the real environment to one area, and aprocessor 140 b configured to control the first and second imagegenerators 110R and 110L to display the image in a representative depth.

The first and second image generators 110R and 110L may generate theimage for the right eye and the image for the left eye, respectively,under control of the processor 140 b. The first and second imagegenerators 110R and 110L are the same as described above, and thus,their detailed descriptions are omitted.

The processor 140 b may not only generate a light modulation signal sothat the first and second image generators 110R and 110L may generatethe images, but may also determine the representative depth from imageinformation or information received from an eye tracking sensor.

The first image convergence member 120R may converge the image for theright eye and the image of the real environment to an area by changingat least one of a light path L₁ of the image for the right eye and alight path L₂ of the image of the real environment. Here, the area maybe a right eye RE of a viewer. The first image convergence member 120Rmay transmit a plurality of rays according to the plurality of lightpaths L₁ and L₂ to the eye of the viewer. The second image convergencemember 120L may converge the image for the left eye and the image of thereal environment to an area by changing at least one of a light path L₃of the image for the left eye and the light path L₂ of the image of thereal environment. Here, the area may be a left eye LE of the viewer.

The first and second image convergence members 120R and 120L may eachinclude the freeform curved surface 121 described above. In addition,the first and second image convergence members 120R and 120L may furtherinclude a waveguide, a light transmissive plate, a beam splitter, atransflective film, etc.

The image transmitted by the light of the first light path L₁ and thethird light path L₃ may be an image provided in an AR device. The imageof the real environment transmitted by the light of the second lightpath L₂ may be an environment faced by the viewer through the AR device.The real environment may include a front view faced by the viewer andmay include a predetermined background subject.

FIG. 11 illustrates an example in which a display apparatus according toan example embodiment is applied to a vehicle. The display apparatus maybe applied to a vehicle head up display apparatus 70. The vehicle headup display apparatus 70 may include an image generator 110 c provided ina region of the vehicle and at least one optical system 120 c configuredto convert a light path for a driver to watch an image generated by theimage generator 110 c. The optical system 120 c may include a freeformoptical system according to an embodiment.

FIG. 12 illustrates an example in which a display apparatus according toan example embodiment is applied to AR glasses or VR glasses. The ARglasses 80 may include an image generator 110 d configured to generatean image and an optical system 120 d configured to guide the image toenter into an eye of a viewer from the image generator 110 d. Theoptical system 120 d may include a freeform optical system according toan embodiment.

In addition, the holographic display apparatuses 10 and 20 according toan embodiment may be realized as various types of wearable devices, headmounted displays (HMDs), glasses-type displays, or goggle-type displays.

The holographic display apparatuses 10 and 20 described above may besynchronized or connected to other electronic devices, such as smartphones, etc., to operate. For example, a processor configured to drivean image generator may be included in the smartphone. In addition, theholographic display apparatuses 10 and 20 described above may beincluded in a smartphone.

According to the holographic display apparatus 10 and 20 and theoperating methods thereof, an optical aberration due to a freeformcurved surface may be compensated for.

According to the holographic display apparatus 10 and 20 and theoperating methods thereof, shape distortion of an image due to thefreeform curved surface may be compensated for.

The holographic display apparatuses 10 and 20 described above may beeasily applied to a wearable device. For example, the holographicdisplay apparatuses 10 and 20 may be applied to a glasses-type ARdisplay device, etc.

It should be understood that embodiments described herein should beconsidered in a descriptive sense only and not for purposes oflimitation. Descriptions of features or aspects within each embodimentshould typically be considered as available for other similar featuresor aspects in other embodiments. While one or more embodiments have beendescribed with reference to the figures, it will be understood by thoseof ordinary skill in the art that various changes in form and detailsmay be made therein without departing from the spirit and scope asdefined by the following claims.

What is claimed is:
 1. A holographic display apparatus comprising: animage generator configured to generate a hologram image by modulatinglight; an optical system comprising a freeform curved surface configuredto form the hologram image generated by the image generator at apredetermined depth; and a processor configured to: generate acomputer-generated hologram (CGH) based on three-dimensional imageinformation using a phase map, the phase map comprising informationabout an optical aberration due to the freeform curved surface; andcontrol the image generator to modulate the light based on the CGH. 2.The holographic display apparatus of claim 1, wherein the phase mapcomprises phase information of the light based on a light path on whichthe light modulated by the image generator is transmitted to a viewer bythe freeform curved surface.
 3. The holographic display apparatus ofclaim 2, wherein the phase information of the light comprises adifference between a phase of the light at a time when the light ismodulated by the image generator and a phase of the light at a time whenthe light reaches the viewer.
 4. The holographic display apparatus ofclaim 2, wherein the phase information is inversely proportional to awavelength of the light modulated by the image generator.
 5. Theholographic display apparatus of claim 2, wherein the phase informationis proportional to a refractive index of a medium on the light path onwhich the light modulated by the image generator is transmitted to theviewer through the freeform curved surface.
 6. The holographic displayapparatus of claim 2, wherein the phase information is proportional to alength of the light path on which the light modulated by the imagegenerator is transmitted to the viewer through the freeform curvedsurface.
 7. The holographic display apparatus of claim 1, wherein theprocessor is further configured to perform phase modulation on the CGHusing the phase map.
 8. The holographic display apparatus of claim 7,wherein the processor is further configured to modulate the CGH using aconjugate value of the phase map.
 9. The holographic display apparatusof claim 1, wherein the processor is further configured to apply, to thethree-dimensional image information, a distortion compensation algorithmconfigured to compensate for image shape distortion due to the freeformcurved surface.
 10. The holographic display apparatus of claim 9,wherein the distortion compensation algorithm comprises an inverselytransformed algorithm of a mapping algorithm, the mapping algorithmbeing configured to map location information of pixels included in thethree-dimensional image information with location information of pixelsof a virtual image corresponding to the three-dimensional imageinformation.
 11. The holographic display apparatus of claim 10, whereinthe processor is further configured to apply the distortion compensationalgorithm to an entirety of the location information of the pixelsincluded in the three-dimensional image information.
 12. The holographicdisplay apparatus of claim 10, wherein the processor is furtherconfigured to apply the distortion compensation algorithm to thelocation information of one or more pixels from among the pixelsincluded in the three-dimensional image information and linearlyinterpolate the location information of the other pixels from among thepixels included in the three-dimensional image information.
 13. Theholographic display apparatus of claim 10, wherein the optical systemcomprises a combiner configured to converge the hologram image andexternal light to one point, the external light corresponding to anexternal environment, and wherein the freeform curved surface isintegral with the combiner.
 14. The holographic display apparatus ofclaim 13, wherein the combiner comprises a waveguide configured totransmit the hologram image, wherein the freeform curved surface isarranged on a surface of the waveguide.
 15. The holographic displayapparatus of claim 13, wherein the combiner further comprises atransflective layer arranged on the freeform curved surface.
 16. Theholographic display apparatus of claim 1, wherein the holographicdisplay apparatus comprises a virtual reality apparatus or an augmentedreality apparatus.
 17. An operating method of a holographic displayapparatus comprising a freeform curved surface, the operating methodcomprising: generating a computer-generated hologram (CGH) based onthree-dimensional image information using a phase map, the phase mapcomprising information about an optical aberration due to the freeformcurved surface; generating a hologram image by modulating light based onthe CGH; and forming the hologram image at a predetermined depth by thefreeform curved surface.
 18. The operating method of claim 17, whereinthe phase map comprises phase information of the light based on a lightpath on which the light modulated as the hologram image is transmittedto a viewer by the freeform curved surface.
 19. The operating method ofclaim 18, wherein the phase information of the light comprises adifference between a phase of the light at a time when the light ismodulated as the hologram image and a phase of the light at a time whenthe light reaches the viewer.
 20. The operating method of claim 18,wherein the generating of the CGH comprises modulating the CGH using aconjugate value of the phase map.